Solving Semi-infinite Linear Programs Using Boosting-Like Methods

Linear optimization problems (LPs) with a very large or even infinite number of constraints frequently appear in many forms in machine learning. A linear program with m constraints can be written as min x∈P n c x with a where I assume for simplicity that the domain of x is the n dimensional probability simplex P n. Optimization problems with an infinite number of constraints of the form a j x ≤ b j , for all j ∈ J, are called semi-infinite, when the index set J has infinitely many elements, e.g. J = R. In the finite case the constraints can be described by a matrix with m rows and n columns that can be used to directly solve the LP. In semi-infinite linear programs (SILPs) the constraints are often given in a functional form depending on j or implicitly defined, for instance by the outcome of another algorithm. In this work I consider several examples from machine learning where large LPs need to be solved. An important case is boosting – a method for combining classifiers in order to improve the accuracy (see [1] and references therein). The most well-known instance is AdaBoost [2]. Under certain assumptions it finds a separating hyperplane in an infinite dimensional feature space with a large margin, which amounts to solving a semi-infinite linear program. The algorithms that I will discuss to solve the SILPs have their roots in the AdaBoost algorithm. The second problem is the one of learning to predict structured outputs, which can be understood as a multi-class classification problem with a large number of classes. Here, every class and example generate a constraint leading to a huge optimization problem [3]. Such problems appear for instance in natural language processing, speech recognition as well as gene structure prediction [4]. Finally, I consider the case of learning the optimal convex combination of kernels for support vector machines [5,6]. I show that it can be reduced to a semi-infinite linear program [7] that is equivalent to a semi-definite programming formulation proposed in [8]. I will review several methods to solve such optimization problems, while mainly focusing on three algorithms related to boosting: LPBoost, AdaBoost * and TotalBoost. They work by iteratively selecting violated constraints while refining the solution of the SILP. The selection of violated constraints is done in a problem dependent manner: a so-called base learning algorithm is employed in …